Web Robot Detection based on Monotonous Behavior

Several studies examined various features on how to most effectively detect web robots. Based on an insight that most web robots, regardless of specifics, would exhibit focused and therefore monotonous behavior, this paper proposes that monitoring the rate of behavioral change is highly effective in detecting sessions initiated by web robots. Empirical evaluation performed on more than one billion requests made to www.microsoft.com web servers confirms that “switching factor” is indeed highly effective. In this paper, we explain the three features whose switching factor was used in web robot detection. Unlike previous studies where the types of web robots to detect were limited to a few types (e.g., text crawlers, link checkers, and email harvester), we extend the types of web robots to image collectors, icon collectors, download tools, etc.